Suspension of Disbelief in Interactive Information Retrieval Evaluation

In literature and art, suspension of disbelief is the willingness of the audience to accept a premise as true even if it is not possible in the real world. Audience members must set-aside their beliefs about the world and accept one or more premises which form the foundation for the new world. Research also requires a suspension of disbelief, especially research conducted in the laboratory. In IIR evaluation, researchers suspend much disbelief. For example, most researchers understand (and believe) that measures developed in the context of system-centered evaluation (e.g., mean average precision) are inappropriate for evaluations involving users, but they still use them anyway. The design and development of IIR systems have been pursued diligently over the years, but less effort and care have been spent on developing IIR evaluation methods and measures. Instead, researchers often use methods and measures that are convenient, even when they are inappropriate and require extraordinary suspension of disbelief. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Copyright 200X ACM X-XXXXX-XX-X/XX/XX ...$5.00. In this talk, I will first describe the IIR study landscape and standard IIR evaluation practices by presenting results of a systematic review of IIR evaluation studies that have been published in 31 sources over the last 40 years (19662006). The following questions will be addressed: How many IIR evaluation studies have been published during this period? How has this number changed over time? Where are IIR evaluations most likely to be published? What have been the most common subjects, tasks, collections and measures used in IIR evaluations? In the second part of the talk, I will discuss problems and limitations of some of the standard ways of doing things and argue that more effort needs to be spent developing, evaluating and using research methods and measures that are tailored to IIR study situations, including IIR system evaluations and studies of IIR user behavior. A certain amount of suspension of disbelief is almost always required for research, but new research efforts can reduce the extent to which this is necessary in IIR. Towards a Methodology for Contextual Information Retrieval Emanuele Di Buccio Department of Information Engineering University of Padua, Italy [email protected] Massimo Melucci Department of Information Engineering University of Padua, Italy [email protected]